Primer on type I and type II errors.

Effective Clinical Practice January/February 1999 Volume 2 Number 1 45 Statistical tests are tools that help us assess the role of chance as an explanation of patterns observed in data.* The most common “pattern” of interest is how two groups compare in terms of a single outcome. After a statistical test is performed, investigators (and readers) can arrive at one of two conclusions: 1) The pattern is probably not due to chance (i.e., the difference is significant) 2) The pattern is likely due to chance (i.e., the difference is not significant. No matter how well the study is performed, either conclusion may be wrong. A type I error is analogous to a false-positive result during diagnostic testing: a difference is shown when in “truth” there is none. Researchers have long been concerned about making this mistake and have conventionally demanded that the probability of a type I error be less than 5%. This convention means that in 100 studies that have yielded positive results, we are willing to accept that in 5 studies these results will be due to chance alone. The probability that a type I error has occurred in a positive study is the exact P value reported.